In any storage system design it is important to intelligently and cost-effectively manage data and meta-data. A cost-effective storage system includes both high and low cost storage devices. An intelligent file transfer policy moves file data between storage devices in the system to relocate infrequently accessed data from feature rich, expensive, highly protected and high-speed disk storage to more economical and possibly slower mass storage in accordance with business goals.
Tiered storage solutions often include a hierarchy of file servers with attached storage of varying cost and capability. The exemplary tiered storage solution of FIG. 1 includes a hierarchy 10 of file servers which serve files and applications to the other clients 12 or other file servers on the network. The hierarchy of Network Attached Server (NAS) includes a primary NAS file server 14 and a secondary NAS file server 16. Each NAS connects directly to a network using TCP/IP over Ethernet. Files are transferred between NAS file servers using any NAS protocol, such as Common Internet File Sharing (CIFS), Network File System (NFS), File Transfer Protocol (FTP) or the like. The storage hierarchy may also include storage for archived data, such as a Content Addressed Storage (CAS) server 20. A Policy Engine (PE) 18 controls the transfer of files between primary NAS 14, secondary NAS 16 and CAS 20.
The PE is a server that retrieves file system meta-data from the primary NAS via various attribute access interfaces including but not restricted to the File Mover Application Interface (API), described in U.S. patent Ser. No. 11/085,898, filed Mar. 21, 2005 and entitled “Selection of Migration Methods Including Partial Read Restore in Distributed Storage Management” by Amegadzie et al, and incorporated herein by reference. The file system meta-data includes a variety of attributes associated with the files in the file system. The Policy Engine (PE) analyzes the value of these attributes in view of policy rules that direct the placement of files at appropriate tiers of the storage hierarchy. Using the File Mover API, the PE issues commands to move data off-line to secondary storage.
An accurate enforcement of a file migration policy can only be performed if the attribute values available to the PE for each file are kept current. File attribute scanning is typically performed prior to policy enforcement to obtain current attribute values. During the scan, each attribute for each file in the file system is read and its value subsequently evaluated with regard to the criteria associated with the policy. File system scanning includes but is not restricted to reading directory contents, looking up file names, and querying the value of file attributes for every file in the NAS server. File attribute values are obtained using multiple NAS operations, and returned to the PE using multiple protocols. The PE then evaluates the retrieved attribute values and other policy related information (such as pathnames, file name extensions, and other available meta-data) to identify files of interest.
The file system scanning method is inefficient in terms of network latency, consumption of network bandwidth and CPU utilization. Thus policy enforcement applications are generally performed during off-peak hours, when client utilization of the storage devices is low. However, as the capabilities and number of files served by the NAS continue to grow, the processing of files for policy enforcement purposes adversely impacts a client's access to the resources. Accordingly, it would be desirable to identify a method and apparatus that would improve the performance of file processing for policy enforcement.